ENT-5725: Added vagrant provider for spawning VMs #156

victormlg · 2025-09-09T15:10:24Z

It was a bit hard to know exactly what approach to take so if you have any better ideas I am interested

This doesn't exactly work yet, because

when creating VMs in a loop with a default name (like dev-1, dev-2, dev-3...) , the vms are created but only the first one created is accessible from the directory where lies the Vagrantfile (I'm using virtualbox)
when creating VMs with a custom name given by an env variable (ike in the pr) the vm is just not accessible, as if it wasn't created, but is actually present in virtualbox.

I am not sure how we should provision the created vms. We could:

just leave them unprovisioned
like I do in the code: let the user input his own provision script
start the VM unprovisioned, then inspect it using nt-discovery.sh and find all the information we need (like what package manager it uses, etc...) then provision it with a modular script using the collected data.

cf_remote/Vagrantfile

craigcomstock

looks like a great start! I have added some notes on things we have solved in jenkins-vms regarding various bits.

cf_remote/Vagrantfile

craigcomstock · 2025-09-10T14:12:04Z

cf_remote/Vagrantfile

+      node.vm.provider "virtualbox" do |vb|
+          vb.memory = VM_MEMORY
+          vb.cpus = VM_CPUS
+          vb.customize [ "guestproperty", "set", :id, "/VirtualBox/GuestAdd/VBoxService/--timesync-set-threshold", 1000 ]


regarding time and dns you might need to include this snippet from jenkins-vms that handles darwin/macos

https://gitlab.com/Northern.tech/CFEngine/jenkins-vms/-/blob/master/scripts/common.rb?ref_type=heads#L104

craigcomstock · 2025-09-10T14:13:23Z

cf_remote/Vagrantfile

+        # previously working Vagrantfiles:
+        # https://fedoraproject.org/wiki/Changes/Vagrant_2.2_with_QEMU_Session#Upgrade.2Fcompatibility_impact
+        v.qemu_use_session = false
+        override.vm.synced_folder "./", "/vagrant", type: :rsync


synced folders are also tricky for older distributions like debian-9:

here are some workarounds for centos-7 and old debians:

https://gitlab.com/Northern.tech/CFEngine/jenkins-vms/-/blob/master/scripts/common.rb?ref_type=heads#L151

cf_remote/main.py

cf_remote/spawn.py

larsewi

Looks great 🚀 Some smaller comments

While reviewing your PR, I never understood what the config argument to ssh and scp is for, or why it's necessary. Could you explain?

cf_remote/commands.py

cf_remote/spawn.py

cf_remote/Vagrantfile

victormlg · 2025-10-02T09:48:02Z

Looks great 🚀 Some smaller comments

While reviewing your PR, I never understood what the config argument to ssh and scp is for, or why it's necessary. Could you explain?

When you spawn a vagrant VM, vagrant generates a ssh config file with a private key and everything set up correctly. You can inspect it by running vagrant ssh-config. When you ssh into a vagrant machine by running vagrant ssh <name>, vagrant is simply doing ssh -F .
Connecting with vanilla ssh was a bit cumbersome because it resulted in a lot of errors and man-in-the-middle attacks warnings and such, so I thought it would be easier to connect by using the ssh config file generated by vagrant

The variable "config" is simply the path to the vagrant-ssh-config file inside the VM's directory. If it is not None, then the connection will connect using it.

larsewi

Looks like there are some failing test :)

victormlg · 2025-10-02T12:35:22Z

synced folders are also tricky for older distributions like debian-9:

here are some workarounds for centos-7 and old debians:

https://gitlab.com/Northern.tech/CFEngine/jenkins-vms/-/blob/master/scripts/common.rb?ref_type=heads#L151

I was thinking we could improve the Vagrantfile and other mechanics, like default provisioning scripts, in future iterations

cf_remote/commands.py

larsewi · 2025-10-02T13:27:01Z

cf_remote/main.py

+        default="aws",
+        choices=["aws", "gcp", "vagrant"],
+    )
+    sp.add_argument("--cpus", help="Number of CPUs of the vagrant instances", type=int)


Consider having a default 1 CPUs

Suggested change

sp.add_argument("--cpus", help="Number of CPUs of the vagrant instances", type=int)

sp.add_argument("--cpus", default=1, help="Number of CPUs of the vagrant instances", type=int)

I can't have a default cpu because I check if "cpus" is defined when validating args, because I don't allow to define the number of cpus for AWS spawned machines

cf_remote/spawn.py

MANIFEST.in

cf_remote/spawn.py

Ticket: ENT-5725 Signed-off-by: Victor Moene <victor.moene@northern.tech>

craigcomstock · 2025-10-06T18:12:31Z

cf_remote/Vagrantfile

+          vb.customize [ "modifyvm", :id, "--uart1", "0x3F8", "4" ]
+          vb.customize [ "modifyvm", :id, "--uartmode1", "file",


what are these doing in here?

I think it is to fix the bug mentioned in the ticket in the comment above (ubuntu/focal64 very slow to boot and reboots once ) . I got that from the Vagrantfile in starter_pack

larsewi · 2025-10-07T07:50:03Z

cf_remote/spawn.py

+        memory = 1024
+    if sync_folder is None:
+        sync_folder = os.getenv("CF_REMOTE_SYNC_ROOT", "/tmp")
+        if sync_folder == "/tmp":


From the VagrantVM.info() function above it looks like the default sync directory is /northern.tech. While here it seems to be /tmp.

I wonder if syncing the /tmp folder can have some security issues. E.g., maybe systemd on both the host machine and the VM starts writing to the same files? Not sure. A good default would be no sync folder at all. You don't really need it if you are using cf-remote to install CFEngine and to deploy policy.

larsewi · 2025-10-07T07:51:37Z

cf_remote/spawn.py

+        sync_folder = os.getenv("CF_REMOTE_SYNC_ROOT", "/tmp")
+        if sync_folder == "/tmp":
+            log.warning(
+                "The synched folder has not been specified. The default synched folder will be set to '/tmp'. \


s/folder/directory/g

larsewi · 2025-10-07T07:56:27Z

cf_remote/spawn.py

+    return [
+        VagrantVM(
+            "{}-{}".format(name, i + 1),
+            "{}.{}".format(start, end % 255),


Do you plan to fix the potential conflicting IP address issue?

larsewi · 2025-10-07T07:58:13Z

cf_remote/ssh.py

+    configs = read_json(SSH_CONFIGS_JSON_FPATH)
+
+    with open(SSH_CONFIG_FPATH, "w") as f:
+        if configs:


Maybe you should do this if before opening the file?

victormlg requested review from craigcomstock and larsewi September 9, 2025 15:10

larsewi reviewed Sep 10, 2025

View reviewed changes

cf_remote/Vagrantfile Outdated Show resolved Hide resolved

craigcomstock reviewed Sep 10, 2025

View reviewed changes

victormlg force-pushed the spawn-vagrant branch 3 times, most recently from 429b367 to 8ff3c80 Compare September 23, 2025 15:37

victormlg force-pushed the spawn-vagrant branch from 8ff3c80 to c96596d Compare September 30, 2025 15:40

victormlg marked this pull request as ready for review September 30, 2025 15:40

victormlg force-pushed the spawn-vagrant branch from c96596d to be78500 Compare September 30, 2025 15:43

victormlg requested a review from larsewi September 30, 2025 15:44

larsewi requested changes Oct 1, 2025

View reviewed changes

victormlg force-pushed the spawn-vagrant branch from be78500 to 73e9bd8 Compare October 2, 2025 11:15

victormlg requested a review from larsewi October 2, 2025 11:16

larsewi requested changes Oct 2, 2025

View reviewed changes

victormlg force-pushed the spawn-vagrant branch from 73e9bd8 to fa2fa4f Compare October 2, 2025 12:30

victormlg requested a review from larsewi October 2, 2025 12:32

victormlg requested a review from craigcomstock October 2, 2025 12:36

larsewi reviewed Oct 2, 2025

View reviewed changes

craigcomstock reviewed Oct 2, 2025

View reviewed changes

cf_remote/spawn.py Show resolved Hide resolved

victormlg force-pushed the spawn-vagrant branch 2 times, most recently from 6af64fc to 1f44cbe Compare October 6, 2025 16:29

victormlg requested a review from larsewi October 6, 2025 16:31

Added vagrant vm spawning provider

3887067

Ticket: ENT-5725 Signed-off-by: Victor Moene <victor.moene@northern.tech>

victormlg force-pushed the spawn-vagrant branch from 1f44cbe to 3887067 Compare October 6, 2025 16:36

craigcomstock approved these changes Oct 6, 2025

View reviewed changes

larsewi reviewed Oct 7, 2025

View reviewed changes

	sp.add_argument("--cpus", help="Number of CPUs of the vagrant instances", type=int)
	sp.add_argument("--cpus", default=1, help="Number of CPUs of the vagrant instances", type=int)

		vb.customize [ "modifyvm", :id, "--uart1", "0x3F8", "4" ]
		vb.customize [ "modifyvm", :id, "--uartmode1", "file",

ENT-5725: Added vagrant provider for spawning VMs #156

Are you sure you want to change the base?

ENT-5725: Added vagrant provider for spawning VMs #156

Uh oh!

Conversation

victormlg commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

craigcomstock left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

larsewi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

victormlg commented Oct 2, 2025

Uh oh!

larsewi left a comment

Choose a reason for hiding this comment

Uh oh!

victormlg commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

victormlg commented Sep 9, 2025 •

edited

Loading

victormlg commented Oct 2, 2025 •

edited

Loading